User interface control based on elbow-anchored arm gestures

ABSTRACT

User interface control based on elbow-anchored arm gestures is disclosed. In one aspect, there is provided a method of user interface control on a computing device based on elbow-anchored arm gestures. A visual user interface (VUI) screen is displayed on a display of a computing device. The VUI screen comprises a plurality of VUI elements arranged in a plurality of VUI element levels. Each VUI element level comprising one or more VUI elements. A spatial location of an elbow of an arm of a user and a spatial location of a wrist of the arm of the user are determined. A three-dimensional (3D) arm vector extending from the spatial location of the elbow to the spatial location of the wrist is then determined. A VUI element in the VUI screen corresponding to the 3D arm vector is then determined based on a predetermined 3D spatial mapping between 3D arm vectors and VUI elements for the VUI screen.

RELATED APPLICATION DATA

The present application claims priority to provisional U.S. patentapplication No. 62/901,737, filed Sep. 17, 2019, the content of which isincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates generally to user interfaces and, morespecifically, to user interface control based on body gestures.

BACKGROUND

User interfaces (UIs) based on body gestures often require users to holdrigid body postures to interact with UI, for example, via mid-air arminteractions. Such postures are impractical in a number of settings,including when sitting relaxed in front of a smart TV, when casuallyinteracting with a system at a distance, or when the user's hands areencumbered. In addition, the performance of mid-air arm interactions maylead to arm fatigue. Accordingly, there exists a need for improvementsin user interface controls based on body gestures.

SUMMARY

The present disclosure provides a user interface control based onelbow-anchored arm gestures. Elbow-anchored arm gestures are gesturesthat primarily involve the forearm (shoulder rotation and elbowflexion/extension) and may be contrasted with full arm gestures.Elbow-anchored arm gestures may be more flexible and which reduce oravoid arm fatigue. In conventional mid-air arms interactions, multiplebody parts are involved and the shoulder movement largely dominates theforces required for moving the arm when perform mid-air arminteractions. Therefore, for reducing arm fatigue when performingmid-air arm interactions, it is desirable to limit the shoulder motion.Known solutions do not consider this biomechanical and ergonomic aspectof mid-air arm interactions and required intensive shoulder musclemovement.

The present disclosure provides methods of user interface control basedon elbow-anchored arm gestures. The method includes determining athree-dimensional arm vector based on the spatial location of an elbowof an arm of a user and the spatial location of a wrist of the arm ofthe user. The determined three-dimensional arm vector is compared andmatched to a corresponding elbow-anchored arm gesture. An executableaction mapped to the matching elbow-anchored arm gesture is determinedand the determined executable action may be performed. The methods areused to control or interact with a user interface including, but notlimited to, a visual user interface (VUI) and VUI elements of VUIs.

The methods and VUIs of the present disclosure are based primarily ondetecting elbow-anchored arm gestures in an interaction space centeredon the user's elbow joint and forearm position and/or motion toaccommodate a range of body postures including arm positions. Thisreduces or eliminates the constraints on body posture associated withconventional mid-air interactions, reduces arm fatigue by reducingshoulder involvement, and reduces the effect of gorilla arm syndromecaused by conventional mid-air interactions. As noted above, knownsolutions do not consider this biomechanical aspect of the mid-air arminteractions and required intensive shoulder muscle movement. Thepresent inventors have recognized that among the three arm joints, userscommonly engage the elbow and wrist when asked to perform mid-air arminteractions. FIG. 14 illustrates the result of an observational studyand shows that a dominant use of elbow and wrist joints, as compared tothe shoulder joint, motions, when users are given free choice to performmid-air arm interactions, such as mid-air arm gestures.

The present disclosure also provides complimentary VUIs having a screenconfiguration and interactions that support the methods of the presentdisclosure. The VUI screen configurations and interactions are based onelbow-anchored arm gestures determined based on the three-dimensionalarm vector with VUI screens configured along a line, such as an arc,formed by arm movement centered at the elbow, i.e. forearm movement. TheVUI screen configuration and interactions may allow more options thanknown mid-air VUI screen configurations. The VUI screens comprise aplurality of VUI elements arranged in a plurality of VUI element levels,each VUI element level comprising one or more VUI elements. A forwardinclination angle between the three-dimensional arm vector and ahorizontal reference plane typically controls the VUI element level thatis selected, and a lateral inclination angle between thethree-dimensional arm vector and a vertical reference plane typicallycontrols the VUI element on the VUI element level that is selected.

The VUI screen configurations are scalable to include more VUI elementsby increasing VUI elements along the horizontal reference plane. Upperlevels of the VUI screens relative to a screen orientation of the VUImay have smaller interaction space as compared to the lower levels ofthe VUI screens relative to a screen orientation of the VUI based on thebiomechanical and ergonomic aspects, therefore upper VUI levels may havefewer VUI elements requiring less horizontal motion as compared to thelower levels. This also helps reduce the false positives due to thenatural arm movement. The VUI screen configurations may also accommodateasymmetries that are known to be latent in arm movements. For example,the forearm side of the VUI screen may have more VUI elements as aresult of increased interaction space compared to the upper arm side.

In accordance with a first embodiment of a first aspect of the presentdisclosure, there is provided a method of user interface control basedon elbow-anchored arm gestures. A visual user interface (VUI) screen isdisplayed on a display of a computing device. The VUI screen comprises aplurality of VUI elements arranged in a plurality of VUI element levels.Each VUI element level comprising one or more VUI elements. A spatiallocation of an elbow of an arm of a user and a spatial location of awrist of the arm of the user are determined based on sensor data. Athree-dimensional (3D) arm vector extending from the spatial location ofthe elbow to the spatial location of the wrist is then determined. A VUIelement in the VUI screen corresponding to the 3D arm vector is thendetermined based on a predetermined 3D spatial mapping between 3D armvectors and VUI elements for the VUI screen.

In some or all examples of the first embodiment of the first aspect,determining the VUI element in the VUI screen corresponding to the 3Darm vector comprises determining a forward inclination angle formedbetween the 3D arm vector and a horizontal reference plane, anddetermining a lateral inclination angle formed between the 3D arm vectorand a vertical reference plane, and determining a VUI element in the VUIscreen based on the forward inclination angle and the lateralinclination angle.

In some or all examples of the first embodiment of the first aspect,determining the VUI element in the VUI screen comprises determining acorresponding VUI element level of the VUI screen among the plurality ofVUI element levels of the VUI screen based on the forward inclinationangle, determining a corresponding VUI element in the determined VUIelement level of the VUI screen based on the lateral inclination angle.

In some or all examples of the first embodiment of the first aspect, themethod further comprises, prior to causing the VUI screen to bedisplayed and in response to input to display the VUI screen,determining a spatial location of an elbow of an arm of a user,determining a spatial location of a wrist of the arm of the user,determining a 3D arm vector extending from the spatial location of theelbow to the spatial location of the wrist, determining an offset anglebetween the 3D arm vector and a vertical centerline from the spatiallocation of the elbow, and generating the VUI screen offset from thecenterline of the display of the computing device in response to adetermination that the offset angle is greater than or equal to athreshold offset angle.

In some or all examples of the first embodiment of the first aspect,upper VUI element levels in VUI screen relative to a screen orientationof the VUI screen have fewer VUI elements than lower VUI element levelsin VUI screen relative to a screen orientation of the VUI screen.

In some or all examples of the first embodiment of the first aspect, theplurality of VUI elements are arranged in spherical grid.

In some or all examples of the first embodiment of the first aspect, themethod further comprises, selecting the determined VUI element, andvisually emphasizing the selected VUI element.

In some or all examples of the first embodiment of the first aspect, themethod further comprises, performing an executable action correspondingto the selected VUI element in response to the detection of confirmationinput.

In some or all examples of the first embodiment of the first aspect, theconfirmation input is a designated hand gesture.

In some or all examples of the first embodiment of the first aspect, themethod further comprises performing an executable action correspondingto the determined VUI element.

In accordance with a second embodiment of the first aspect of thepresent disclosure, there is provided a method of user interface controlon a computing device based on elbow-anchored arm gestures. A spatiallocation of an elbow of an arm of a user and a spatial location of awrist of the arm of the user are determined based on sensor data. Athree-dimensional (3D) arm vector extending from the spatial location ofthe elbow to the spatial location of the wrist is then determined. Anexecutable action corresponding to the 3D arm vector is then determinedbased on a predetermined 3D spatial mapping between 3D arm vectors andexecutable actions.

In some or all examples of the first embodiment of the first aspect, themethod further comprises performing the executable action correspondingto the 3D arm vector.

In accordance with a third embodiment of the first aspect of the presentdisclosure, there is provided a method of detecting selection of aparticular menu item user interface element in a user interfaceincluding a plurality of level user interface elements and including aplurality of menu item user interface elements. The method includescausing presentation, on a display, of the user interface, detecting aspatial location of an elbow of an arm of a user, detecting a spatiallocation of a wrist of the arm of the user and defining athree-dimensional arm vector extending from the spatial location of theelbow to the spatial location of the wrist. The method further includesdetermining a forward inclination angle formed between thethree-dimensional arm vector and a horizon, determining, based on theforward inclination angle, a current level user interface element amongthe plurality of level user interface elements, detecting a directionfor a lateral displacement of the wrist and based on the direction,detecting selection of the particular menu item user interface element,where the particular menu item user interface element is associated withthe direction and the current level user interface element.

In accordance with a second aspect of the present disclosure, there isprovided a computing device comprising a memory and a processor systemcomprising at least one processor coupled to the memory. Thenon-transitory machine-readable medium has tangibly stored thereonexecutable instructions for execution by the processor system of thecomputing device. The executable instructions, in response to executionby the processor system, cause the processor system to perform themethods described above and herein.

In accordance with a second aspect of the present disclosure, there isprovided a non-transitory machine-readable medium having tangibly storedthereon executable instructions for execution by a processor system of acomputing device. The processing system comprising at least oneprocessor, wherein the executable instructions, in response to executionby the processor system, cause the processor system to perform themethods described above and herein.

Other aspects and features of the present disclosure will becomeapparent to those of ordinary skill in the art upon review of thedescription of specific implementations of the present disclosure inconjunction with the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computing device suitable for practicingthe teachings of the present disclosure.

FIG. 2 illustrates an arm of a user, which may be detected by sensors ofthe computing device in FIG. 1 in accordance with the presentdisclosure.

FIGS. 3A and 3B illustrate a three-dimensional arm vector in athree-dimensional space.

FIG. 4 illustrates the interaction space of the user as a subregion on asphere with the elbow at the center of the sphere.

FIG. 5 illustrates the interaction space of the user and a cartographicprojection of the interaction space of the user.

FIG. 6 is a flowchart illustrating a method of user interface controlbased on elbow-anchored arm gestures in accordance with one embodimentof the present disclosure.

FIG. 7 is a flowchart illustrating a method of user interface controlbased on elbow-anchored arm gestures in accordance with anotherembodiment of the present disclosure.

FIGS. 8A-8D illustrate visual user interface screens in accordance withembodiments of the present disclosure.

FIG. 9 is a schematic representation illustrating the relationshipbetween the forward inclination angle and VUI element level inaccordance with embodiments of the present disclosure.

FIG. 10 illustrates a visual user interface screen in accordance withanother embodiment of the present disclosure.

FIG. 11 illustrates a visual user interface screen in accordance with afurther embodiment of the present disclosure.

FIG. 12 illustrates a visual user interface screen in accordance with afurther embodiment of the present disclosure.

FIG. 13A illustrates a spherical visual user interface in accordancewith one embodiment of the present disclosure.

FIG. 13B illustrates a spherical visual user interface in accordancewith another embodiment of the present disclosure.

FIG. 14 is a table demonstrating the results of an observational studyon the joint motions preferred by users when given free choice toperform arm gestures.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

The present disclosure is made with reference to the accompanyingdrawings, in which embodiments are shown. However, many differentembodiments may be used, and thus the description should not beconstrued as limited to the embodiments set forth herein. Rather, theseembodiments are provided so that this application will be thorough andcomplete. Wherever possible, the same reference numbers are used in thedrawings and the following description to refer to the same elements,and prime notation is used to indicate similar elements, operations orsteps in alternative embodiments. Separate boxes or illustratedseparation of functional elements of illustrated systems and devicesdoes not necessarily require physical separation of such functions, ascommunication between such elements may occur by way of messaging,function calls, shared memory space, and so on, without any suchphysical separation. As such, functions need not be implemented inphysically or logically separated platforms, although they areillustrated separately for ease of explanation herein. Different devicesmay have different designs, such that although some devices implementsome functions in fixed function hardware, other devices may implementsuch functions in a programmable processor with code obtained from amachine-readable medium. Lastly, elements referred to in the singularmay be plural and vice versa, except where indicated otherwise eitherexplicitly or inherently by context.

The term “gesture” is used in the present disclosure and is intended toinclude positions as well as movements or motions. The presentdisclosure also refers to VUI screens.

Reference is first made to FIG. 1 which illustrates a computing device102 suitable for practicing the teachings of the present disclosure. Thecomputing device 102 may be a multi-purpose or special purposeelectronic device. Examples of the computing device 102 include, but arenot limited to, a smart TV, a personal computer such as a desktop orlaptop computer, a smartphone, tablet, a personal camera or cameraperipheral, smart glasses or other head device mounted smart display, asmart speaker or other smart or IoT (Internet of Things) device such asa smart appliance, among other possibilities.

The computing device 102 includes a processing system comprising aprocessor 104 (such as a microprocessor or central processing unit(CPU)) which controls the overall operation of the computing device 102.The processing system may include one or more other types of processorscoupled to the processor 104, such as a graphic processing unit (GPU), atensor processing unit (TPU), a neural processing unit (NPU), anapplication specific integrated circuit, or a field programmable gatearray (FPGA), for offloading certain computing tasks. The processor 104is coupled to a plurality of components via a communication bus (notshown) which provides a communication path between the components andthe processor 104. The processor 104 is coupled to Random Access Memory(RAM) 108, Read Only Memory (ROM) 110, and persistent (non-volatile)memory 112 such as flash memory and a communication subsystem 130. Thecommunication subsystem 130 includes one or more wireless transceiversfor exchanging radio frequency signals with wireless networks. Thecommunication subsystem 130 may also include a wireline transceiver forwireline communications with wired networks. The wireless transceiversmay include one or a combination of Bluetooth transceiver or othershort-range wireless transceiver, a Wi-Fi or other wireless local areanetwork (WLAN) transceiver for communicating with a WLAN via a WLANaccess point (AP), or a wireless wide area network (WWAN) transceiversuch as a cellular transceiver for communicating with a radio accessnetwork (e.g., cellular network). The cellular transceiver maycommunicate with any one of a plurality of fixed transceiver basestations of the cellular network within its geographic coverage area.The wireless transceivers may include a multi-band cellular transceiverthat supports multiple radio frequency bands. Other types of short-rangewireless communication include near field communication (NFC), IEEE802.15.3a (also referred to as UltraWideband (UWB)), Z-Wave, ZigBee,ANT/ANT+ or infrared (e.g., Infrared Data Association (IrDA)communication). The wireless transceivers may include a satellitereceiver for receiving satellite signals from a satellite network thatincludes a plurality of satellites which are part of a global orregional satellite navigation system.

The computing device 102 also comprises a display 138 coupled to theprocessor 104. The display 138 may be implemented as a conventionaldisplay screen. However, there are many alternative manners in which toprovide visual feedback to the user. In the context of augmented realityand/or virtual reality visual feedback mechanisms, visual feedback maybe provided through projection on the user's retina. Furthermore, thecomputing device 102 may arrange the display 138 as a projection onto asurface.

The computing device 102 may also comprise sensor in the form a camera136 and a detection and ranging (DAR) unit 122 such as a LiDAR unit. Thecomputing device 102 may also comprise a microphone 132, a speaker 134,and a satellite receiver 140 for receiving satellite signals from asatellite network each coupled to the processor 104, depending on thetype of the computing device 102. The computing device 102 may alsocomprise one or more other input devices 142 such as a touchscreen,keyboard, keypad, navigation tool, buttons, switches or dials dependingon the type of the computing device 102. A touchscreen may be providedas the display 138.

The computing device 102 may also comprise a plurality of additionalsensors 120 coupled to the processor 104. The sensors 120 may comprisean accelerometer, a motion sensor, a gyroscope, an inertial measurementunit (IMU), a proximity sensor, an orientation sensor, electroniccompass or altimeter, among other possibilities.

A computer vision system may be provided by the camera 136 and/or a DARunit 122 in combination with a computer vision application 172. Thecomputer vision system may be based on one or a combination of imagesfrom the camera 136 and point cloud positional data from the DAR unit122. The computer vision system is configured to detect the bodyposition of a user, including a spatial location of an elbow of an armof the user and a spatial location of a wrist of the arm of the user. Inone embodiment, the camera captures a video of a user performingelbow-anchored gestures and determines the three-dimensional arm vectorand optionally a type of elbow-anchored gesture performed using knownimage processing techniques. The computer vision system receives imagesof the user's arm and hand and processes the images using known imageprocessing techniques to detect the location of the elbow and wrist, anddetermine the three-dimensional arm vector and track thethree-dimensional arm vector across frames to determine that anelbow-anchored gesture was performed. In some embodiments, the computervision system determines three-dimensional coordinates of the tworelevant body joints (i.e., elbow and wrist). From the three-dimensionalcoordinates, an arm length is derived, and azimuth and inclination aredetermined based on conversion equations known in the art. From theazimuth and inclination, the arm length may be mapped to x and ycoordinates of a VUI displayed on the display 138 using a linear mappingequation known in the art.

Alternatively, the computing device 102 may be wirelessly coupled to asmart device 200 via the communication subsystem 130 that comprises oneor more dedicated sensors configured to detect a spatial location of anelbow of an arm of a user (referred to as “elbow sensors” 124) such asan acceleration sensor (e.g., accelerometer) and gyroscope, and one ormore dedicated sensors configured to detect a spatial location of awrist of the arm of the user (referred to as “wrist sensors” 126) suchas an acceleration sensor (e.g., accelerometer) and gyroscope. In someembodiments, the smart device 200 is a wearable device such as a glove,sleeve, band or similar garment to be worn by the user, and the elbowsensors 124 and wrist sensors 126 are embedded therein. In otherembodiments, the smart device 200 comprises a smart watch comprising anaccelerometer and gyroscope worn by a user at their wrist and anaccelerometer and gyroscope are carried by a separate sensor module (notshown) at the elbow of the user which wirelessly communicates with thesmart device 200. The wrist sensors 126 and elbow sensors 124 of thesmart device 200 capture the acceleration and rotation of the wrist andelbow and transmit signals indicative of acceleration and rotation ofthe user's wrist and elbow to computing device 102 via the communicationsubsystem 130 via Bluetooth™ or other suitable wireless communicationprotocol. In yet other embodiments, a single sensor at the wrist in asmart watch or the like may be used. In such embodiments, the elbow andwrist location are not known. An orientation of the sensor and the smartwatch relative to an arbitrary origin. It is assumed that theorientation of the watch is the same orientation of the arm and theelbow is at rest on an arm rest or the like and static duringinteraction. A calibration may be performed during an initial setup. Inall embodiments, the computing device 102 processes the signals todetermine the three-dimensional arm vector mentioned above.

Operating system software 150 executable by the processing system,including the processor 104 is stored in the persistent memory 112 butmay be stored in other types of memory devices, such as ROM 108 orsimilar storage element. The operating system software 150 renders a VUIfor user interaction on the display 138 of the computing device 102. Auser may interact with the VUI elements rendered on the display 138 viaelbow-anchored gestures and/or input devices 142 as described in furtherdetail below. A number of application programs 152 executable by theprocessing system, including the processor 104 are also stored in thepersistent memory 112. The application programs 152 comprises a gesturecontrol application 170 and a computer vision application 172.Alternatively, the gesture control application 170 and computer visionapplication 172 may be part of the operating system software 150, suchas part of the VUI, or an application programming interface (API). Thegesture control application 170 comprises instructions for userinterface control based on elbow-anchored arm gestures in accordancewith the teachings of the present disclosure, such as the methods 600and 700 described below.

The memory 112 stores a variety of data 154, including sensor dataacquired by the plurality of sensors 120, including sensor data acquiredby the sensors 120. The memory 112 also stores input data 156 acquiredby the display 138 and/or other input devices 142, user data includinguser preferences, settings and possibly biometric data about the userfor authentication and/or identification, a download cache includingdata downloaded via the wireless transceivers, and saved files. Systemsoftware, software modules, specific device applications, or partsthereof, may be temporarily loaded into RAM 108. Communication signalsreceived by the computing device 102 may also be stored in RAM 108.Although specific functions are described for various types of memory,this is merely one embodiment, and a different assignment of functionsto types of memory may be used in other embodiments.

The computing device 102 may also comprise a battery (not shown) as apower source, such as one or more rechargeable batteries that may becharged, for example, through charging circuitry coupled to a batteryinterface such as the serial data port. The battery provides electricalpower to at least some of the components of the computing device 102,and the battery interface (not shown) provides a mechanical andelectrical connection for the battery. Alternatively, an AC power sourcemay be provided.

FIG. 2 illustrates an arm 202 of a user. The sensors 120 of thecomputing device 102 may detect a spatial location 204 of an elbow ofthe arm 202 and a spatial location 206 of a wrist of the arm 202. Thespatial location 204 of the elbow and the spatial location 206 of thewrist may be understood to define a three-dimensional arm vector 208extending from the spatial location 204 of the elbow to the spatiallocation 206 of the wrist. A feature of the three-dimensional arm vector208 is a forward inclination angle 212, which may be considered to existbetween the three-dimensional arm vector 208 and a horizontal referenceplane 210. Although the horizontal reference plane 210 is selected herefor a reference plane for the forward inclination angle 212, it may beunderstood that any consistent reference plane may be used.

FIGS. 3A and 3B illustrate the three-dimensional arm vector 208 in athree-dimensional space defined X, Y and Z axes. The forward inclinationangle 212 is formed between the three-dimensional arm vector 208 and ahorizontal reference plane 210 defined by the XZ plane. A lateralinclination angle formed between the three-dimensional arm vector 208and a vertical reference plane 210 defined by the YZ plane. The YZ planemay be defined by, or coincident with, a vertical centerline of theuser's elbow from the spatial location of the elbow. A coordinate systemwhich may either be with reference to the body orientation or the earthis used. The origin of the coordinate system is placed at the elbowjoint and the centerline may be defined as the Y axis parallel to thebody/earth orientation, as described in more detail below. A user seatedin a chair and resting their elbow on a surface (i.e. an arm of a chair)is able to move his or her arm through the XZ plane, YZ plane or a both,even while in a seated position. The user may rest his or her below onan arm rest of a chair, sofa or the like, or any other horizontalsurface, while performing elbow-anchored arm gestures and still move hisor her arm through one or both of the XZ and YZ planes, thereby avoidingarm fatigue.

Since the distance between the wrist and the elbow joint is always thelength of the forearm, the hand motion may be defined by a spherecentered at the elbow joint O with radius the length of the forearmshown in FIG. 4. Only a subsection of the sphere is accessible due tothe range of motion of the elbow. To describe regions on the sphere thatare reachable (the region bound by the line EGAFCE in FIG. 4), fourplanes are considered: ODE, ODF, OEF, and the plane that crosses G and Aand is perpendicular to z. Without loss of generality, assuming the useris right handed, the plane ODE corresponds to the rightmost limit theuser is able to move the hand to, the plane ODF corresponds to theuser's chest, which sets the left boundary. In an observational study itwas found that participants performed gestures with the hand raisedabove a certain level, which may be approximated by the plane OEF forconsistency and simplicity. On a couch, chair or other seated position,this plane may represent the armrest, for instance.

In this setting, the maximum angle range of the angle EOF of theleft-right forearm movement is around 100°, with points E and F beingthe bottom left and bottom right corners of the input space. The anglerange of the angle COD for the up-down movement is around 70° where thepoint C is on the sphere and bisects angle LEOF. The points G and A are,respectively, the top right and top left corners of the space. Acoordinate system may be set that aligns the x-axis with OC and z-axiswith OD. This results in angles COE=−60°, COF=40°, and FOA=EOG=70°. Thisallows any point in the interaction space to be specified by two anglesθ∈[−60°, 40°], and ϕ∈[0°, 70°], which are referred to as azimuth andinclination, analogous to the longitude-latitude geographic coordinatesystem on Earth.

The interaction space (also known as motor space) of the user isillustrated in FIG. 5. The interaction space is the three-dimensionalspace in which the user is able to move his or her arm while the user'selbow is anchored on a surface, such as an arm of a chair, sofa, and thelike, based on biomechanics, for example.

For interaction with VUIs, the three-dimensional interaction space ofthe user is spatially mapped to a two-dimensional plane of the VUI. Thethree-dimensional interaction space may be projected onto thetwo-dimensional plane. The projection, referred to as a cartographicprojection or map projection, is a representation of all or part of thethree-dimensional interaction space in a two-dimensional plane havingthe same dimension as the display 138. The three-dimensional arm vectorsand elbow-anchored arm gestures in three-dimensions detected by theprocessor 104 may be mapped to VUI elements in two-dimensions using apredetermined spatial mapping between the three-dimensional (3D)elbow-anchored arm gestures and vectors and the two-dimensional (2D) VUIelements. The term “elbow-anchored arm gestures” includes arm positionsas well as arm movements.

The interaction space may be based on a user in a seated position insome embodiments, for example, the “reachable” space when the spatiallocation 204 of the elbow is fixed. If an imaginary sphere is definedwith a center at the spatial location 204 of the elbow, the interactionspace may be defined as a portion of a surface of the imaginary spheredefined by the limits of lateral forearm motion and vertical forearmmotion. The location of elbow joint is center of the sphere and the handis on the surface of the sphere. The forearm length defines the radiusof the sphere in the three-dimensional space.

Mapping the motor space to the VUI may involve the following steps insome embodiments. The spatial location 204 of the elbow and spatiallocation of the wrist are detected. These define a sphere centered atthe spatial location 204 of the elbow with the hand on the surface ofthe sphere. The body orientation of the user is detected of predefined.The body orientation is used to define three planes indicating the left,right and the bottom boundaries of the interaction space. Each planecontains the center of the sphere. This interaction space is subdividedinto vertical levels using planes cutting through the y-axis. Theseplanes are the base XY plane rotated around the y-axis and separated byequal angles. An example of this is shown in FIG. 9. The rotation anglecorresponds to the y-coordinate on the display 138 or x-coordinate ifthe VUI is scrollable. The interaction space is further subdivided byplanes parallel to the XZ plane, separated by equal arc length,resulting in cells of similar area and shape. The arc length correspondsto x-coordinate on the display 138. To map any interaction spacecoordinate (X0, Y0, Z0) to a display coordinate, a rotation angle thetais used, where theta=arctan (Z0/X0) as a y-coordinate on the display 138and arctan (X0*cos(theta)/Y0). Optionally, 10 degree margins may beprovided on each side for easier reachability and better control. EachVUI element in a plurality of VUI elements correspond to a cell in aplurality of cells in the interaction space.

FIG. 6 is a flowchart illustrating a method of user interface controlbased on elbow-anchored arm gestures in accordance with one embodimentof the present disclosure. The method 600 is used to control or interactwith a user interface including, but not limited to, a VUI. The method600 may be carried out by software such as the gesture controlapplication 170 executed, for example, by at least the processor 104 ofthe processing system of the computing device 102 illustrated in FIG. 1.

At operation 604, the processor 104 determines a spatial location 204for the elbow of the arm 202 of the user and a spatial location 206 forthe wrist of the arm 202 based on signals received from the wrist sensor124 and the elbow sensor 126.

At operation 608, the processor 104 determines a three-dimensional armvector 208 extending from the spatial location 204 of the elbow to thespatial location 206 of the wrist.

At operation 610, the processor 104 determines an executable actioncorresponding to the three-dimensional arm vector 208 based on apredetermined 3D spatial mapping between three-dimensional arm vectorsand executable actions. Alternatively, an executable action may be basedon an elbow-anchored arm gesture, for example defined by a trajectoryover k past historical locations, rather than a single three-dimensionalarm vector.

The executable actions may be any suitable type of executable actions.The particular executable actions may vary based on the type of thecomputing device 102, the active application or context, among otherfactors. The type of executable actions which may vary based on the typeof UI of the computing device 102. For UI without a VUI component, suchas a smart TV controlled by motion gestures, examples of executableactions include “power on”, “power off”, “increase volume”, “decreasevolume”, “mute”, “settings”, “home screen”, or “start video streamingservice”, the meaning of which would be understood to persons skilled inthe art. For VUIs, the executable action may comprise selection of a VUIelement within a VUI screen displayed on the display 138.

At operation 612, the processor 104 causes the determined executableaction corresponding to the three-dimensional arm vector 208 to beperformed.

FIG. 7 is a flowchart illustrating a method of user interface controlbased on elbow-anchored arm gestures in accordance with one embodimentof the present disclosure. The method 700 is used to control or interactwith a VUI. The method 700 is similar to the method 600 except that theexecutable actions in method 700 related to selecting VUI elements in aVUI screen. The method 700 may be carried out by software such as thegesture control application 170 executed, for example, by at least theprocessor 104 of the processing system of the computing device 102illustrated in FIG. 1.

At operation 702, the processor 104 causes the display of a VUI screenon the display 138 of the computing device 102. The VUI screen includesa plurality of VUI elements. The plurality of VUI elements are arrangedin a plurality of VUI element levels (or rows), with each VUI elementlevel (or merely “level”) including one or more VUI elements. ExampleVUI screen configurations are described below. The processor 104 maycause the display of a VUI screen to be displayed based on inputreceived by the user, such as a voice input received by the microphone132 and converted to a command by speech-to-text synthesis and speechrecognition, a button press on a remote control device or other inputdevice 142, or a gesture detected by the wrist sensor 126, by processingimages captured by the camera 136 of the user's arm and hand using thecomputer vision system to identify a gesture performed by the user andto determine whether the identified gesture is an activation gesture.For example, the VUI may be invoked in response to detection of aspecific hand gesture or hand orientation, such as a hand anchoredperpendicular for a threshold duration. The manner of activating orinvoking the VUI screen is outside the scope of the present disclosure.

At operation 604, the processor 104 determines a spatial location 204for the elbow of the arm 202 of the user and a spatial location 206 forthe wrist of the arm 202 based on signals received from the wrist sensor124 and the elbow sensor 126.

At operation 608, the processor 104 determines a three-dimensional armvector 208 extending from the spatial location 204 of the elbow to thespatial location 206 of the wrist.

At operation 710, the processor 104 determines a VUI element in the VUIscreen corresponding to the three-dimensional arm vector 208 based on apredetermined 3D spatial mapping between three-dimensional arm vectorsand VUI elements for the VUI screen. In some examples, the 3D spatialmapping may map each three-dimensional arm vector in a plurality ofthree-dimensional arm vectors to range of forward inclination angles andlateral inclination angles. In such examples, the operation 710 maycomprise a number of sub-operations based on analysing the forwardinclination angle and lateral inclination angle of the determinedthree-dimensional arm vector, as described below. Alternatively, a VUIelement in the VUI screen may be based on an elbow-anchored arm gesture,for example defined by a trajectory over k past historical locations,rather than a single three-dimensional arm vector.

At operation 712, the processor 104 determines a forward inclinationangle 212 formed between the three-dimensional arm vector 208 and ahorizontal reference plane 210 defined by the XZ plane. At operation714, the processor 104 determines a lateral inclination angle formedbetween the three-dimensional arm vector 208 and a vertical referenceplane 210. At operation 716, the processor 104 determines a VUI elementin the VUI screen based on the forward inclination angle formed betweenthe three-dimensional arm vector 208 and a horizontal reference plane210 defined by the XZ plane and the lateral inclination angle formedbetween the three-dimensional arm vector 208 and a vertical referenceplane 210. In the present example, the operation 716 comprisessub-operations 718 and 720 based on analysing the forward inclinationangle and lateral inclination angle of the determined three-dimensionalarm vector. At operation 718, the processor 104 determines acorresponding level of the VUI screen among the plurality of levels ofthe VUI screen based on the forward inclination angle. Alternatively,the processor 104 may determine the corresponding level of the VUIscreen among the plurality of levels of the VUI screen based on avertical height of the three-dimensional arm vector. At operation 720,the processor 104 determines a corresponding VUI element in thedetermined VUI level of the VUI screen based on the lateral inclinationangle. Alternatively, the processor 104 may determine the correspondingVUI element in the determined VUI level of the VUI screen based on alateral distance of the three-dimensional arm vector from the YZ plane.It will be appreciated that the forward inclination angle 212 typicallycontrols the VUI element level that is selected, and that the lateralinclination angle typically controls the VUI element on the VUI elementlevel that is selected.

FIG. 9 is a schematic representation illustrating the relationshipbetween the forward inclination angle and VUI element level inaccordance with embodiments of the present disclosure. Each VUI elementlevel in a VUI screen may be associated with a range of angle values.Range may be the same for each VUI element level in the VUI screen.Using FIG. 9 as an example, each VUI element level may have a range ofangle values spanning 18 degrees (90 degrees divided by 5) so that aforward inclination angle of between 0 and 18 degrees corresponds to VUIelement level 1, a forward inclination angle of between 19 and 36degrees corresponds to VUI element level 2, a forward inclination angleof between 37 and 54 degrees corresponds to VUI element level 3, aforward inclination angle of between 55 and 72 degrees corresponds toVUI element level 1, and a forward inclination angle of between 73 and90 degrees corresponds to VUI element level 5.

At operation 722, in response to determining the corresponding VUIelement in the determined VUI level of the VUI screen, the determinedVUI element is selected by the processor 104 as the current or activeVUI element. As noted below, separate confirmation input is required toperform an executable action associated with the selected VUI element.The use of separate confirmation input provides a form of VUI lockingthat inhibits accidentally/unintentionally changing the VUI elementlevel when performing further elbow-anchored arm gestures with lateralmovements. Depending on the VUI screen configuration, this may beomitted.

The selection-based VUI locking may be performed automatically when theuser performs a lateral motion at a VUI element level from a center orinitial position. The selection-based VUI locking does not require anactivation gesture. The selection-based locking mechanism may only beprovided when there are multiple items at a given VUI element level. Ifthere are only two VUI elements per VUI element level, for example withone each side of the user's arm, crossing the centerline be used toselect a VUI element and execute the executable action associated withselected VUI element without the visually emphasizing the selected VUIelement and providing confirmation input regarding the same. However, ifthere are multiple VUI elements on each side of the centerline at givenVUI element level, crossing to select or focus on the VUI element andconfirmation input, such as a hand gesture, is typically used to selectthe VUI element and execute the executable action associated with it.

At operation 724, in response to selecting the determined VUI element asthe current or active VUI element, the processor 104 causes the selectedVUI element to be visually emphasized to provide feedback to the user.The visual emphasis enhances the determined VUI element, therebydistinguishing the determined VUI element from the remainder of the VUIelements in the VUI screen. The visual emphasis by be caused by changinga color of the determined VUI element, the size of the determined VUIelement (e.g., enlarging the determined VUI element), or focusing thedetermined VUI element with onscreen indicator such as a caret, cursor,halo or pointer. Audio feedback in the form of a designated sound ortone or may also be provided to the user when a VUI element is selected.

Reference will be briefly made to FIG. 8A, which illustrates oneembodiment of a VUI screen 800 displayed on the display 138 of thecomputing device 102 in accordance with the present disclosure. The VUIscreen 800 comprises a plurality of VUI elements. The VUI screen 800comprises four levels of VUI elements denoted 810-1, 810-2, 810-3 and810-4, respectively. Each VUI element levels includes two VUI elements:a left side and right side VUI element spatially associated with theleft and right sides of the VUI screen 800, respectively. The first VUIelement level 820-1 comprises a first left VUI element 820-1L and afirst right VUI element 820-1R. The second VUI element level 820-2comprises a second left VUI element 820-2L and a second right VUIelement 820-2R. The third VUI element level 820-3 comprises a third leftVUI element 820-3L and a third right VUI element 820-3R. The fourth VUIelement level 820-4 comprises a third left VUI element 820-4L and athird right VUI element 820-4R. In the shown example, the second leftVUI element 810-2L is visually emphasized by an onscreen indicator 830.The VUI element levels are given ordinals (first, second, third, fourth)only to distinguish the levels from each other rather than to define anorder. It may be assumed that the user's arm begins a given interactionat rest and that the first VUI element that is selected by the user andvisually emphasized in the given interaction may be different than thesecond left VUI element 810-2L, and may depend on the speed of theuser's movements and the responsiveness to the computing device 102 indetecting gestures.

At operation 726, the processor 104 determines whether confirmationinput to perform an executable action associated with the selected VUIelement has been received/detected.

When the processor 104 determines that confirmation input to perform anexecutable action associated with the selected VUI element has beenreceived/detected, operations proceed to operation 728 at which theprocessor 104 causes the executable action corresponding to the selectedVIU element to be performed. The confirmation input may be any suitableinput such as a designated hand gesture identified by the computervision system 122, voice input, or other input received via a buttonpress of a remote control device or other input device 142 of thecomputing device 102. The designated hand gesture may be, for example, apinch gesture in the form of a pinching action with the user's fingers,a closing of the user's palm or a “dwell” gesture in the form ofmaintaining the arm position at a given position (within tolerance) fora duration of time equal to, or greater than, a threshold duration.

When the processor 104 determines that confirmation input to perform anexecutable action associated with the selected VUI element has not beenreceived/detected, operations proceed to operation 730 at which theprocessor 104 determines whether the three-dimensional arm vector haschanged by an amount equal to, or greater than, a threshold amount. Forexample, the processor 104 may determine whether the forward angle orlateral inclination angle of the three-dimensional arm vector haschanged by an amount equal to, or greater than, a threshold amount. Itwill be appreciated that operation 730 comprises operations similar tooperations 604, 606 and 608 to determine a three-dimensional arm vector.The threshold amount may vary based on the previously determinedthree-dimensional arm vector. For example, the threshold amount may besmaller at upper VUI element levels of the VUI screen relative of ascreen orientation of the VUI screen and larger forward inclinationangles.

When the processor 104 determines that the three-dimensional arm vector208 has changed by an amount equal to, or greater than, the thresholdamount, operations return to operation 710 at which the processor 104determines a new or changed VUI element in the VUI screen correspondingto the three-dimensional arm vector 208 based on the predetermined 3Dspatial mapping.

When the processor 104 determines that the three-dimensional arm vector208 has not changed by an amount equal to, or greater than, thethreshold amount, operations return to operation 726.

Although not shown, the processor 104 monitors for and determines whenthe methods 600 and 700 are to be ended, for example, by receivingcorresponding input to close the VUI screen displayed in operation 702.

In alternative embodiments, rather than selecting and visuallyemphasising the selected VUI element in operation 724, the processor 104may cause the executable action corresponding to the three-dimensionalarm vector 208 to be performed without selecting and visuallyemphasising the selected VUI element.

FIG. 8B illustrates an alternative VUI screen 803. The VUI screen 803 issimilar to the VUI screen 800 in FIG. 8A except that each level of theVUI also includes a center VUI element in each level, identifiedindividually as first central VUI element 820-1C, second central VUIelement 820-2C, third central VUI element 820-3C and fourth central VUIelement 820-4C.

FIGS. 8C and 8D illustrates VUI screens 805A and 805B respectively. TheVUI screens 805A and 805B are angled or tilted relative to the VUIscreens 800 and 803 of FIGS. 8A and 8B. The three-dimensional arm vector208 at the start of a given interaction has an offset angle greater thana threshold offset angle, an angled or tilted VUI screen may bedisplayed. This is an optional feature. A determination as to whetherthe three-dimensional arm vector 208 has an offset angle greater than orequal to a threshold offset angle is made by the processor 104 at thestart of an interaction, before the VUI screen is displayed in operation702. The threshold offset angle may be 5 degrees, 10 degrees or 15degrees in some examples. FIG. 8C illustrates a VUI screen titled to theleft whereas FIG. 8D illustrates a VUI screen titled to the right. Anidentification of arm used by the active (or current) user may bepredetermined and stored in the settings in the memory 112 of thecomputing device 102. Alternatively, a preliminary determination of thearm being used by the active user may be made by the processor 104before the determination as to whether the three-dimensional arm vector208 has an offset angle greater than the threshold offset angle.

The offset angle may be measured between the three-dimensional armvector and a vertical centerline from the spatial location of the elbow.An orientation or tilt angle (e.g., amount of tilt) of the VUI screenmay be dynamically determined and the layout of the VUI screendetermined based on the angularly difference between thethree-dimensional arm vector 208 and reference. The use of angled ortilted VUI screens based on an initial elbow angle allows the VUI toadapt to the available interaction space of the user's elbow andaccommodates a plurality of different user arm positions, for example,while the user is seated.

It will be appreciated that a user has more control over the lateralinclination angle when the forward inclination angle is small and viceversa. Accordingly, upper levels of the VUI screens relative to a screenorientation of the VUI may have smaller interaction space as compared tothe lower levels of the VUI screens relative to a screen orientation ofthe VUI based on the biomechanical and ergonomic aspects. Thus, toaccommodate the available interaction space, upper VUI levels may havefewer VUI elements requiring less horizontal motion as compared to thelower levels. This also helps reduce the false positives due to thenatural arm movement. A pyramidal VUI configuration accommodates suchbiomechanical aspects. Alternatively, the number of VUI elements mayremain the same but the mapping is adapted such that a different rangeof motion (or angular range for the three-dimensional arm vector) ateach VUI element level is matched to same number of VUI elements.

FIG. 10 illustrates a visual user interface screen 1000 in accordancewith another embodiment of the present disclosure having a three-levelpyramidal configuration. The top level has a left VUI element 1020-1L1and a right VUI element 1020-1R1. The middle level has two left VUIelements 1020-2L1, 1020-2L2 and two right VUI elements 1020-2R1,1020-2R2. The bottom level has three left VUI elements 1020-3L1,1020-3L2, 1020-3L3 and three right VUI elements 1020-3R1, 1020-3R2,1020-3R3. The onscreen selection indicator is indicated by reference1030.

FIG. 11 illustrates a visual user interface screen 1100 in accordancewith a further embodiment of the present disclosure. The visual userinterface screen 1100 provides a three-level pyramidal configuration ofdifferent granularity. A top level includes a top left user interface1110-TL and a top right user interface 1110-TR. A middle level includes,on a left side, a middle first left user interface 1110-M1L and a middlesecond left user interface 1110-M2L, and on a right side, a middle firstright user interface 1110-M1R and a middle second right user interface1110-M2R. A bottom level includes, on a left side, a bottom first leftuser interface 1110-B1L, a bottom second left user interface 1110-B2Land a bottom third left user interface 1110-B3L, and on a right side, abottom first right user interface 1110-B1R, a bottom second right userinterface 1110-B2R and a bottom third right user interface 1110-B3R. Theonscreen selection indicator is indicated by reference 1130.

FIG. 12 illustrates a visual user interface screen 1200 in accordancewith another embodiment of the present disclosure. The VUI screen 1200is a asymmetrical and accommodate asymmetries that are known to belatent in arm movements. Elbow muscle allows a wide range of inwardmovement, therefore a VUI screen may be asymmetrical with more VUIelements on the inner side in contrast to outward (away from body). Inother words, forearm side of the VUI screen may be configured to havemore VUI elements as a result of increased interaction space compared tothe upper arm side. The top level has a VUI element 1220-1L. The secondlevel has two VUI elements 1220-2L1, 1220-2L2. The third level has threeVUI elements 1220-3L1, 1220-3L2, 1220-3L3. The fourth and bottom levelhas four VUI elements 1220-4L1, 1220-4L2, 1220-4L3 and 1220-4L3. Theonscreen selection indicator is indicated by reference 1230. To inhibitthe false selection when the user brings their down hand back to therest position, a locking mechanism may be used which does not allow theVUI level to be changed while a VUI element is selected as describedabove.

FIG. 13A illustrates a spherical visual user interface 1300 having inaccordance with one embodiment of the present disclosure. The sphericalvisual user interface 1300 may be provided as part of a VUI screen thatmay be used in combination with the methods of the present disclosure.The spherical visual user interface 1300 comprises a sphere or ellipsoidin which the surface is divided into a plurality of VUI elementsreferred to by the reference number 1305, providing a spherical orellipsoidal grid. The VUI elements may be equally sized ordifferentially sized depending on the embodiment. The VUI elements maybe any shape projected onto a sphere such as squares, rectangles, trap,hexagon or circles. A VUI element 1305 in the spherical visual userinterface 1300 may be selected using arm gestures, based on athree-dimensional arm vector, as described above. The currently selectedVUI element, as referred to as the “focus”, may be visually emphasizedby an onscreen selection indicator 1310. A position, orientation andzoom level of the spherical visual user interface 1300 may be adjustedby pan, rotation and zoom operations, which are be performed in responseto detection of corresponding input, such as a designated hand gesturedetected by the computer vision system 122, voice input, or other inputreceived via a button press of a remote control device or other inputdevice 142 of the computing device 102. The spherical visual userinterface 1300 may have an infinite radius, which would make the VUI aflat surface, similar to a rectangular VUI.

An executable action associated with the selected VUI element may becaused to be performed by confirmation input. The confirmation input maybe any suitable input such as a designated hand gesture detected by thecomputer vision system 122, voice input, or other input received via abutton press of a remote control device or other input device 142 of thecomputing device 102. The designated hand gesture may be, for example, apinch gesture in the form of a pinching action with the user's fingers,a closing of the user's palm or a “dwell” gesture in the form ofmaintaining the arm position at a given position (within tolerance) fora duration of time equal to, or greater than, a threshold duration.

FIG. 13B illustrates a spherical visual user interface 1320 having inaccordance with another embodiment of the present disclosure. Thespherical visual user interface 1320 is similar to the spherical visualuser interface 1300 except that the position of the “focus” or onscreenselection indicator 1330, is fixed. The currently selected VUI elementor “focus”, may be changed by rotating the spherical visual userinterface 1320 using arm gestures, which may be based on rotations ofthe three-dimensional arm vector 208, causing additional content to berevealed. An executable action associated with the selected VUI elementmay be caused to be performed by confirmation input similar to thespherical VUI 1300 described above.

The spherical VUIs 1300 and 1320 may be suitable used for any kind ofgrid-like UIs such as galleries used in streaming media applications, orspherical/3D mapping applications.

The teachings of the present disclosure may be applied to smart TVs,smart speakers and other computing devices with which mid-airinteractions are used, including virtual and augmented reality computingdevices. The teachings of the present disclosure may be extended to userinterfaces without a VUI, such as a purely motion based UI, withsuitable adaption of the teachings of the present disclosure toaccommodate for the lack of a dedicated display as the case may be.

General

The steps and/or operations in the flowcharts and drawings describedherein are for purposes of example only. There may be many variations tothese steps and/or operations without departing from the teachings ofthe present disclosure. For instance, the steps may be performed in adiffering order, or steps may be added, deleted, or modified, asappropriate.

The coding of software for carrying out the above-described methodsdescribed is within the scope of a person of ordinary skill in the arthaving regard to the present disclosure. Machine-readable codeexecutable by one or more processors of one or more respective devicesto perform the above-described method may be stored in amachine-readable medium such as the memory of the data manager. Theterms “software” and “firmware” are interchangeable within the presentdisclosure and comprise any computer program stored in memory forexecution by a processor, comprising Random Access Memory (RAM) memory,Read Only Memory (ROM) memory, EPROM memory, electrically EPROM (EEPROM)memory, and non-volatile RAM (NVRAM) memory. The above memory types areexamples only, and are thus not limiting as to the types of memoryusable for storage of a computer program.

All values and sub-ranges within disclosed ranges are also disclosed.Also, although the systems, devices and processes disclosed and shownherein may comprise a specific plurality of elements, the systems,devices and assemblies may be modified to comprise additional or fewerof such elements. Although several example embodiments are describedherein, modifications, adaptations, and other implementations arepossible. For example, substitutions, additions, or modifications may bemade to the elements illustrated in the drawings, and the examplemethods described herein may be modified by substituting, reordering, oradding steps to the disclosed methods.

Features from one or more of the above-described embodiments may beselected to create alternate embodiments comprised of a subcombinationof features which may not be explicitly described above. In addition,features from one or more of the above-described embodiments may beselected and combined to create alternate embodiments comprised of acombination of features which may not be explicitly described above.Features suitable for such combinations and subcombinations would bereadily apparent to persons skilled in the art upon review of thepresent disclosure as a whole.

In addition, numerous specific details are set forth to provide athorough understanding of the example embodiments described herein. Itwill, however, be understood by those of ordinary skill in the art thatthe example embodiments described herein may be practiced without thesespecific details. Furthermore, well-known methods, procedures, andelements have not been described in detail so as not to obscure theexample embodiments described herein. The subject matter describedherein and in the recited claims intends to cover and embrace allsuitable changes in technology.

Although the present disclosure is described at least in part in termsof methods, a person of ordinary skill in the art will understand thatthe present disclosure is also directed to the various elements forperforming at least some of the aspects and features of the describedmethods, be it by way of hardware, software or a combination thereof.Accordingly, the technical solution of the present disclosure may beembodied in a non-volatile or non-transitory machine-readable medium(e.g., optical disk, flash memory, etc.) having stored thereonexecutable instructions tangibly stored thereon that enable a processingdevice to execute examples of the methods disclosed herein.

The term “processor” may comprise any programmable system comprisingsystems using microprocessors/controllers or nanoprocessors/controllers,central processing units (CPUs), neural processing units (NPUs), tensorprocessing units (TPUs), hardware accelerators, digital signalprocessors (DSPs), application specific integrated circuits (ASICs),field-programmable gate arrays (FPGAs) reduced instruction set circuits(RISCs), logic circuits, and any other circuit or processor capable ofexecuting the functions described herein. The term “database” may referto either a body of data, a relational database management system(RDBMS), or to both. As used herein, a database may comprise anycollection of data comprising hierarchical databases, relationaldatabases, flat file databases, object-relational databases, objectoriented databases, and any other structured collection of records ordata that is stored in a computer system. The above examples are exampleonly, and thus are not intended to limit in any way the definitionand/or meaning of the terms “processor” or “database”.

The present disclosure may be embodied in other specific forms withoutdeparting from the subject matter of the claims. The described exampleembodiments are to be considered in all respects as being onlyillustrative and not restrictive. The present disclosure intends tocover and embrace all suitable changes in technology. The scope of thepresent disclosure is, therefore, described by the appended claimsrather than by the foregoing description. The scope of the claims shouldnot be limited by the embodiments set forth in the examples, but shouldbe given the broadest interpretation consistent with the description asa whole.

The invention claimed is:
 1. A method of user interface control on acomputing device based on elbow-anchored arm gestures, comprising:causing a visual user interface (VUI) screen to be displayed on adisplay of the computing device, wherein the VUI screen comprises aplurality of VUI elements arranged in a plurality of VUI element levels,each VUI element level comprising one or more VUI elements; determining,based on sensor data, a spatial location of an elbow of an arm of auser; determining, based on sensor data, a spatial location of a wristof the arm of the user; determining a three-dimensional (3D) arm vectorextending from the spatial location of the elbow to the spatial locationof the wrist; and determining a VUI element in the VUI screencorresponding to the 3D arm vector based on a predetermined 3D spatialmapping between 3D arm vectors and VUI elements for the VUI screen. 2.The method of claim 1, wherein determining the VUI element in the VUIscreen corresponding to the 3D arm vector comprises: determining aforward inclination angle formed between the 3D arm vector and ahorizontal reference plane; determining a lateral inclination angleformed between the 3D arm vector and a vertical reference plane; anddetermining a VUI element in the VUI screen based on the forwardinclination angle and the lateral inclination angle.
 3. The method ofclaim 2, wherein determining the VUI element in the VUI screencomprises: determining a corresponding VUI element level of the VUIscreen among the plurality of VUI element levels of the VUI screen basedon the forward inclination angle; and determining a corresponding VUIelement in the determined VUI element level of the VUI screen based onthe lateral inclination angle.
 4. The method of claim 1, furthercomprising: prior to causing the VUI screen to be displayed: in responseto input to display the VUI screen: determining a spatial location of anelbow of an arm of a user; determining a spatial location of a wrist ofthe arm of the user; determining a 3D arm vector extending from thespatial location of the elbow to the spatial location of the wrist;determining an offset angle between the 3D arm vector and a verticalcenterline from the spatial location of the elbow; and generating theVUI screen offset from the centerline of the display of the computingdevice in response to a determination that the offset angle is greaterthan or equal to a threshold offset angle.
 5. The method of claim 1,wherein upper VUI element levels in VUI screen relative to a screenorientation of the VUI screen have fewer VUI elements than lower VUIelement levels in VUI screen relative to a screen orientation of the VUIscreen.
 6. The method of claim 1, wherein the plurality of VUI elementsare arranged in spherical grid.
 7. The method of claim 1, furthercomprising: selecting the determined VUI element; and visuallyemphasizing the selected VUI element.
 8. The method of claim 7, furthercomprising: performing an executable action corresponding to theselected VUI element in response to the detection of confirmation input.9. The method of claim 8, wherein the confirmation input is a designatedhand gesture.
 10. The method of claim 1, further comprising: performingan executable action corresponding to the determined VUI element.
 11. Acomputing device, comprising: a display; a processor coupled thedisplay, wherein the processor is configured to: cause a visual userinterface (VUI) screen to be displayed on the display, wherein the VUIscreen comprises a plurality of VUI elements arranged in a plurality ofVUI element levels, each VUI element level comprising one or more VUIelements; determine, based on sensor data, a spatial location of anelbow of an arm of a user based on sensor data; determine, based onsensor data, a spatial location of a wrist of the arm of the user;determine a three-dimensional (3D) arm vector extending from the spatiallocation of the elbow to the spatial location of the wrist; anddetermine a VUI element in the VUI screen corresponding to the 3D armvector based on a predetermined 3D spatial mapping between 3D armvectors and VUI elements for the VUI screen.
 12. The computing device ofclaim 11, wherein the processor is configured to determine the VUIelement in the VUI screen corresponding to the 3D arm vector by:determining a forward inclination angle formed between the 3D arm vectorand a horizontal reference plane; determining a lateral inclinationangle formed between the 3D arm vector and a vertical reference plane;and determining a VUI element in the VUI screen based on the forwardinclination angle and the lateral inclination angle.
 13. The computingdevice of claim 12, wherein the processor is configured to determine theVUI element in the VUI screen by: determining a corresponding VUIelement level of the VUI screen among the plurality of VUI elementlevels of the VUI screen based on the forward inclination angle; anddetermining a corresponding VUI element in the determined VUI elementlevel of the VUI screen based on the lateral inclination angle.
 14. Thecomputing device of claim 11, wherein the processor is furtherconfigured to: prior to causing the VUI screen to be displayed: inresponse to input to display the VUI screen: determine a spatiallocation of an elbow of an arm of a user; determine a spatial locationof a wrist of the arm of the user; determine a 3D arm vector extendingfrom the spatial location of the elbow to the spatial location of thewrist; determine an offset angle between the 3D arm vector and avertical centerline from the spatial location of the elbow; and generatethe VUI screen offset from the centerline of the display of thecomputing device in response to a determination that the offset angle isgreater than or equal to a threshold offset angle.
 15. The computingdevice of claim 11, wherein upper VUI element levels in VUI screenrelative to a screen orientation of the VUI screen have fewer VUIelements than lower VUI element levels in VUI screen relative to ascreen orientation of the VUI screen.
 16. The computing device of claim11, wherein the plurality of VUI elements are arranged in sphericalgrid.
 17. The computing device of claim 11, wherein the processor isfurther configured to: select the determined VUI element; and visuallyemphasize the selected VUI element.
 18. The computing device of claim11, wherein the processor is further configured to: perform anexecutable action corresponding to the selected VUI element in responseto the detection of confirmation input.
 19. The computing device ofclaim 11, wherein the processor is further configured to: perform anexecutable action corresponding to the determined VUI element.
 20. Anon-transitory machine-readable medium having tangibly stored thereonexecutable instructions that, in response to execution by a processor ofa computing device, cause the processor to: cause a visual userinterface (VUI) screen to be displayed on a display of the computingdevice, wherein the VUI screen comprises a plurality of VUI elementsarranged in a plurality of VUI element levels, each VUI element levelcomprising one or more VUI elements; determine, based on sensor data, aspatial location of an elbow of an arm of a user; determine, based onsensor data, a spatial location of a wrist of the arm of the user;determine a three-dimensional (3D) arm vector extending from the spatiallocation of the elbow to the spatial location of the wrist; anddetermine a VUI element in the VUI screen corresponding to the 3D armvector based on a predetermined 3D spatial mapping between 3D armvectors and VUI elements for the VUI screen.